Skip to content

ci: build validation datasets with three speakers - #118

Merged
jsugg merged 1 commit into
mainfrom
ci/synthetic-dataset-actor-count
Nov 10, 2026
Merged

jsugg merged 1 commit into
mainfrom
ci/synthetic-dataset-actor-count

Conversation

@jsugg

@jsugg jsugg commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Summary

Every hosted validation lane (darwin-x86_64, macos15-mps, linux-3.13-cli, full-dataset quality gate) has failed on main since the training readiness contract landed in #85, at training_readiness_insufficient_class_support before training starts. Control runs on main: 42761198900 (quality gate), 42761250998 (linux-3.13).

Root cause

scripts/build_synthetic_ravdess_dataset.py defaults to --actors 1 2. Readiness (_split_findings in ser/_internal/models/training_readiness.py) requires a speaker-disjoint train/dev/test split (split_utterances_three_way) with min_remaining_per_class_split support per class in every partition.

Actor-count derivation

Each synthetic actor records all 8 emotions once, so a class's support in a partition equals that partition's actor count. Default policy (schema.py): test_size=0.25, dev_size=0.10, min_remaining_per_class_split=1; the quality gate sets SER_TEST_SIZE=0.25, the same value.

  • Test cut, GroupShuffleSplit over N speakers: n_test = ceil(0.25 N), leaving M = N - n_test.
  • Dev cut over M speakers at 0.10 / 0.75: n_dev = ceil(0.133 M); grouping needs at least 2 speakers (dataset_splitting.py, can_group_by_speaker).
  • N=2: M=1, so the dev cut cannot group, leaving empty classes plus speaker overlap. That is the 3 findings seen in CI.
  • N=3: test 1, dev 1, train 1, so every class has support 1 in every partition.

Empirically, the real readiness check on builder output gives 3 blocking findings for 2 actors and 0 for 3 or 4 actors. The minimum is 3.

Change

  • All 7 builder invocations across 5 workflows pass --actors 1 2 3 explicitly. The builder default is unchanged: no test or doc invokes the builder, and the only actors: 2 fixtures are unrelated transcription-coverage data.
  • Regression test tests/suites/integration/models/test_synthetic_validation_dataset_readiness.py: parses every workflow's builder arguments, builds each dataset, and runs the real run_training_readiness under the default split policy. Against the previous workflows it fails with the same findings CI reported.

No readiness thresholds or checks were changed.

Validation

Local gates: make fmt lint type prepush-check ci-contracts test-cov (1209 passed, 79.98%). Lanes dispatched on this branch: results are posted below.

The training readiness contract requires a speaker-disjoint
train/dev/test split with minimum per-class support in every
partition. With the default split policy (test 0.25, dev 0.10,
minimum 1 per class) GroupShuffleSplit needs three speakers: one
held out for test, leaving two so the dev cut can still group by
speaker. The builder default of two actors left the dev split empty,
so every hosted lane failed readiness before training.

Workflows now pass --actors explicitly; the builder default is left
alone. A regression test builds each lane dataset and runs the real
readiness contract.
@jsugg

jsugg commented Nov 10, 2026

Copy link
Copy Markdown
Owner Author

Lane results on this branch (dispatched with workflow defaults):

Controls on main before this change failed at readiness: 42761198900, 42761250998.

@jsugg
jsugg merged commit c6823d2 into main Nov 10, 2026
17 checks passed
@jsugg
jsugg deleted the ci/synthetic-dataset-actor-count branch November 10, 2026 06:45
jsugg added a commit that referenced this pull request Nov 10, 2026
ci: build validation datasets with three speakers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant